Portable computing device installed in or mountable to a shopping cart

ABSTRACT

A computing device installed in or mountable to a shopping cart, includes a camera configured to capture an image, and a processor configured to acquire commodity data of a registered commodity, when an object is placed in the shopping cart, perform object recognition on an image of the object captured by the camera, and determine whether the recognized object is a commodity corresponding to the acquired commodity data, and upon determining that the recognized object is not the commodity corresponding to the acquired commodity data, issue a request for displaying a message instructing registration of the recognized object as a registered commodity.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/910,731, filed Mar. 2, 2018, which is based upon and claims thebenefit of priority from Japanese Patent Application No. 2017-041920,filed Mar. 6, 2017, the entire contents of which are incorporated hereinby reference.

FIELD

Embodiments described herein relate generally to a portable computingdevice installed in or mountable to a shopping cart.

BACKGROUND

In retail stores such as supermarkets, checkout systems allowingcustomers to perform payment processing have been introduced to achieveefficiency at the point of sale (POS). For example, self-servicecheckout systems including self-service checkout terminals have beenintroduced. In the self-service checkout systems, commodities areregistered by reading commodity codes (generally, barcodes) throughcustomer operations, payment information of the registered commoditiesis generated, and then processing of the payment for the registeredcommodities is performed.

Further, in recent years, personal checkout systems in which commodityregistration is performed on commodities planned to be purchased whilecustomers are shopping, have been considered. In such systems,commodities inserted in carts or baskets are identified as commoditiesplanned to be purchased.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a system configuration diagram schematically illustrating anoverall configuration of a system according to an embodiment.

FIG. 2 is a block diagram illustrating configurations of electronicapparatuses according to the embodiment.

FIG. 3 is a diagram illustrating an example of a clip terminal mountedonto a cart according to the embodiment.

FIG. 4 is a diagram illustrating an exterior of the clip terminalaccording to the embodiment.

FIG. 5 is a flowchart illustrating an operation of the electronicapparatus according to the embodiment.

FIG. 6 is a flowchart illustrating an operation of the clip terminalaccording to the embodiment.

FIG. 7 is a flowchart illustrating the operation of the clip terminalaccording to the embodiment.

FIG. 8 is a diagram illustrating a commodity checklist according to theembodiment.

FIG. 9A is a diagram illustrating an example of a message according tothe embodiment.

FIG. 9B is a diagram illustrating an example of another messageaccording to the embodiment.

FIG. 9C is a diagram illustrating an example of another messageaccording to the embodiment.

FIG. 10 is a diagram illustrating an example of an image captured by acamera according to the embodiment.

FIG. 11 is a diagram illustrating examples of objects corresponding tocommodities detected from the image illustrated in FIG. 10.

FIG. 12 is a flowchart illustrating registration location determinationperformed to determine whether confirmation is necessary, according tothe embodiment.

FIG. 13 is a diagram illustrating an example of layout of commodityshelves and a position of a cart in a store.

DETAILED DESCRIPTION

Embodiments provide a portable computing device installed in ormountable to a shopping cart that is linked with a hand-held commodityregistration apparatus to reliably register commodities to be purchasedthrough customer operation.

According to an embodiment, a computing device installed in or mountableto a shopping cart, includes a camera configured to capture an image,and a processor configured to acquire commodity data of a registeredcommodity, when an object is placed in the shopping cart, perform objectrecognition on an image of the object captured by the camera, anddetermine whether the recognized object is a commodity corresponding tothe acquired commodity data, and upon determining that the recognizedobject is not the commodity corresponding to the acquired commoditydata, issue a request for displaying a message instructing registrationof the recognized object as a registered commodity.

Hereinafter, the embodiment will be described with reference to thedrawings.

FIG. 1 is a system configuration diagram schematically illustrating anoverall configuration of a system according to an embodiment. In thesystem illustrated in FIG. 1, a system 10 installed in a headquarter anda store system 12 installed in each of a plurality of stores areconnected via a network 14 including a dedicated line or the Internet.

In the system 10 installed in the headquarter, for example, aheadquarter system 10A, a customer system 10B, a payment system 10C, asales promotion system 10D, and the like are connected to each other viaa network 10E such as a local area network (LAN).

The headquarter system 10A performs a process of controlling the storesystem installed in each store. The customer system 10B managescustomers who purchase commodities using the store system. The paymentsystem 10C performs a process (individual authentication, creditpayment, or the like) for payment using a credit card or the like by acustomer in the store system. The sales promotion system 10D performsmanagement through the store system to provide customers (in particular,electronic apparatuses used by customers) with information regardingcommodity advertisement for sales promotion, a store guide, or the like.

An example of the store system 12 in a store may be a point of sales(POS) system. In the store system 12, for example, a store server 20A, aclip terminal management server 20B, a plurality of accounting machines20C, a plurality of POS terminals 20D, and the like are installed andconnected to each other via a network 20E such as a LAN.

The store server 20A controls the store system 12. The store server 20Astores data regarding commodities sold in the store, for example, datasuch as a commodity code of each commodity, a commodity name, a unitprice, a weight, a commodity position sold in the store (e.g., theposition of a commodity shelf), and a commodity image. In commodityregistration, data of a commodity corresponding to a commodity code readby scanning the commodity can be received from the store server 20A andpayment information (including an amount of money, a commodity, and anumber) can be generated. The clip terminal management server 20Bmanages an electronic apparatus (e.g., a clip terminal 50 to bedescribed below). The clip terminal 50 is used to check whethercommodities registered using an electronic apparatus 30 as commoditiesplanned to be purchased are identical to commodities inserted in thecart 60 or a basket as commodities planned to be purchased. Theaccounting machine 20C performs payment processing on a commodityregistered using the electronic apparatus 30 through a customeroperation. The POS terminal 20D performs registration of a commodity ofa customer through a salesclerk operation and payment processing on theregistered commodity.

A plurality of access points (e.g., base stations) 20F are connected tothe network 20E wired in the store. The access point 20F performswireless communication with various electronic apparatuses used in thestore via, for example, a wireless LAN. The electronic apparatuses usedin the store includes, for example, the electronic apparatus 30 used bythe customer, an electronic apparatus 40 used by a salesclerk, and theclip terminal 50 mounted on the cart 60.

For example, a smartphone and a tablet personal computer (PC) are usedas the electronic apparatuses 30 and 40. The electronic apparatus 30cooperates with the clip terminal 50 to perform registration ofcommodities planned to be purchased based on an application program(including a check program) for using services provided by the storesystem. 12 and presence checking performed to check whether thecommodities registered through commodity registration are identical tocommodities inserted in the cart 60 or the basket as commodities plannedto be purchased. The electronic apparatus 40 performs confirming so thata salesclerk (attendant) can easily confirm whether commodityinformation registered is identical to the commodities inserted in thecart 60 based on a confirmation program for confirming the commodityregistration performed by the electronic apparatus 30.

The electronic apparatus 30 and the clip terminal 50 may communicate viathe clip terminal management server 20B (via the network 20E) or maycommunicate directly through short-range wireless communication.Further, the electronic apparatuses may be connected via a communicationcable to perform wired communication.

FIG. 2 is a block diagram illustrating configurations of the electronicapparatus 30 and the clip terminal 50 according to the embodiment. Thedescription will be made of a smartphone as an example of the electronicapparatus 30. The clip terminal 50 is assumed to be configured as adedicated clip terminal 50 that executes presence checking in the storesystem 12. Hereinafter, the clip terminal 50 will be described. The clipterminal 50 can also be realized by installing the check program on adedicated apparatus (e.g., a computer) such as a smartphone on which acamera is mounted.

The electronic apparatus 30 includes a processor 30A, a memory 30B, astorage device 30C, a touch panel 30D, a camera 30E, a communicationmodule 30F, and a battery 30G.

The processor 30A is a circuit that controls each unit by executing aprogram recorded on the memory 30B to carry out various processes of theelectronic apparatus 30.

The memory 30B records data to be used for the processor 30A to executevarious processes in addition to programs to be executed by theprocessor 30A. The programs recorded on the memory 30B include acommodity registration program executing registration of commoditiesplanned to be purchased and a check program executing presence checkingperformed to check whether commodities registered through the commodityregistration are identical to commodities inserted in the cart 60 or abasket as commodities planned to be purchased. The data stored in thememory 30B includes payment information generated based on a commoditycode read during the commodity registration and commodity informationregarding a commodity registered through the commodity registration.

The storage device 30C may be a hard disk drive (HDD) device, asolid-state drive (SSD) device, or the like. The storage device 30Crecords data to be used for the processor 30A to perform variousprocesses or data generated by a process of the processor 30A.

The touch panel 30D is a device on which an input operation is performedby a customer and includes a display device and a touch sensor. Anyscreen such as a graphical user interface (GUI) screen associated with aprocess of the processor 30A is displayed on the touch panel 30D.

The camera 30E captures an image under the control of the processor 30A.

The communication module 30F is, for example, a wireless communicationinterface, and controls communication with an external electronicapparatus under the control of the processor 30A. The communicationmodule 30F can control communication such as wireless communication withthe access point 20F and short-range wireless communication with anotherelectronic apparatus such as the clip terminal 50.

The clip terminal 50 includes a processor 50A, a memory 50B, a storagedevice 50C, a touch panel 50D, a camera 50E, a communication module 50F,a battery 50G, an I/O interface 50H, and a reader-writer 50J.

The processor 50A is a circuit that controls each unit by executing aprogram recorded on the memory 50B to realize various processes of theclip terminal 50.

The memory 50B records data to be used for the processor 50A to performvarious processes in addition to programs to be executed by theprocessor 50A. The programs recorded on the memory 50B include a checkprogram for executing presence checking performed to check whethercommodities registered through the registration are identical tocommodities inserted in the cart 60 or a basket as commodities plannedto be purchased. The check program includes an image processing programwhich is executed to perform image processing to identify a commodityfrom an image captured by the camera 50E and detect that the identifiedcommodity is inserted in the cart 60 or taken out from the cart 60. Thedata stored in the memory 50B includes various kinds of data such ascommodity information registered through commodity registration usingthe electronic apparatus 30, object management data (see FIG. 11) formanaging an object detected through image processing, and a commoditychecklist (see FIG. 8) for checking commodities inserted in the cart 60are stored.

The storage device 50C may be a hard disk drive (HDD) device, asolid-state drive (SSD) device, or the like. The storage device 50Crecords data to be used for the processor 50A to perform variousprocesses or data generated by a process of the processor 50A.

The touch panel 50D is a device on which an input operation is performedby a customer and includes a display device and a touch sensor. Anyscreen such as a graphical user interface (GUI) screen associated with aprocess of the processor 50A is displayed on the touch panel 50D.

The camera 50E captures an image under the control of the processor 50A.

The communication module 50F controls communication with an externalelectronic apparatus under the control of the processor 50A. Thecommunication module 50F can control communication such as wirelesscommunication with the access point 20F and short-range wirelesscommunication with another electronic apparatus such as the electronicapparatus 30.

The I/O interface 50H is an interface for inputting a signal from anexternal apparatus and is, for example, a gravimetric sensor 601 (e.g.,a scale, a pressure sensitive instrument, a gravimeter, or the like)mounted on a basket bottom 600 of the cart 60. The gravimetric sensor601 detects the weight of a commodity inserted (put) in the cart 60 andoutputs a detection signal.

The reader-writer 50J communicates with a radio frequency (RF) tagmounted on each commodity to be sold in the store or a commodity shelfor the like in the store. A communication range by the reader-writer 50Jis for example, a communicable range with an RF tag of a commodityinserted in the cart 60 or a communicable range with an RF tag mountedon a commodity shelf when the cart 60 is moved in the store. Thereader-writer 50J can be used to detect that a commodity is inserted inor take output from the cart 60 or can be used to determine a locationof the cart 60 on which the clip terminal 50 is mounted, in the store.

FIG. 3 is a diagram illustrating an example of the clip terminal 50mounted onto the cart 60 according to the embodiment. FIG. 4 is adiagram illustrating an exterior of the clip terminal 50 according tothe embodiment.

For example, the clip terminal 50 is mounted on an upper frame 60B ofthe cart 60 used in the store. As illustrated in FIG. 4, a clip 60 k isinstalled in the clip terminal 50 so that mounting can be performedtherewith. Accordingly, it is not necessary to mount the clip terminal50 on all of the carts 60 in advance in the store. The clip terminal 50may be mounted only when the cart 60 is for a customer using a serviceof presence checking. Therefore, large-scale investment such asremodeling of the store or exchange of the carts 60 is not necessary inorder to implement the presence checking service. Therefore, thisservice can be provided at a low cost.

The example illustrated in FIG. 3 is an example in which the clipterminal 50 is mounted on an upper frame 60B near a handle 60A of thecart 60. When the clip terminal 50 is mounted, the camera 50E providedin the clip terminal 50 faces a basket bottom 60C and an imaging rangeof the camera 50E is adjusted so that at least all of the commoditiesinserted in the cart can be imaged. The imaging range of the camera 50Epreferably contains the upper frame 60B so that areas outside and insidethe basket can be clearly distinguished in an image captured by thecamera 50E. Further, the imaging range of the camera 50E is preferablyset such that a motion of for example, an arm (hand) of a customer canbe imaged to detect a customer operation of inserting and taking out acommodity.

As illustrated in FIG. 4, the clip terminal 50 is mounted on the upperframe 60B so that a display surface of the touch panel 50D faces acustomer. Thus, it is easy to confirm information (e.g., a message orthe like) indicating an image captured by the camera 50E or a checkresult of the presence checking performed using the captured image.

Next, operations of the electronic apparatus 30 and the clip terminal 50used in the store system 12 according to the embodiment will bedescribed. FIG. 5 is a flowchart illustrating an operation of theelectronic apparatus 30 according to the embodiment. FIGS. 6 and 7 areflowcharts illustrating an operation of the clip terminal 50 accordingto the embodiment.

When a customer performs commodity registration using the electronicapparatus 30, an application program for using services provided by thestore system 12 is activated. Hereinafter, the processor 30A of theelectronic apparatus 30 executes the application program to performcommodity registration and presence checking.

First, the electronic apparatus 30 performs login for connection to thestore system 12 (Act10). In the login process, for example, a barcodecontaining a membership code unique to a member and attached to amembership card is read by the camera 30E. The processor 30A extractsthe membership code from the barcode read by the camera 30E andtransmits the membership code to the clip terminal management server20B. The clip terminal management server 20B performs memberauthentication by comparing with membership data managed by the customersystem 10B based on the membership code received from the electronicapparatus 30.

When the electronic apparatus 30 is notified that the memberauthentication is successful by the clip terminal management server 20B,the electronic apparatus 30 displays a selection screen of servicesavailable in the store. The services available in the store includedisplay of leaflet information, guide display of special commodities,display of a store guide (map), and use of the clip terminal 50, inaddition to commodity registration for purchase planning. When using theclip terminal 50, a message is output from the clip terminal 50 insituations where commodity registration is not performed on, forexample, a commodity inserted in the cart 60. Accordingly, a customercan be prevented from forgetting to register a commodity planned to bepurchased. While a customer is shopping in the store, a commodityplanned to be purchased is checked by the clip terminal 50. Therefore,it is possible to reduce a burden on an attendant performingconfirmation of registered commodities after commodity registration bythe customer ends.

When a customer uses the clip terminal 50, the store system 12 mayprompt the customer to use the clip terminal 50 by providing a benefit(e.g., reward points or the like) to the customer. The data indicatingthe benefit for the customer is stored in the customer system 10B.

When the clip terminal 50 is used, the electronic apparatus 30 and theclip terminal 50 communicate with each other. For example, the clipterminal 50 is mounted on the cart 60 used by a customer (or thecustomer is allowed to use the cart 60 on which the clip terminal 50 ismounted in advance), and a link between the electronic apparatus 30 ofthe customer and the clip terminal 50 is established. For example, theelectronic apparatus 30 and the clip terminal 50 are paired to be ableto communicate with each other through short-range wirelesscommunication. Alternatively, a unique code of the clip terminal 50 isread by the electronic apparatus 30 and is transmitted from theelectronic apparatus 30 to the clip terminal management server 20B. Theclip terminal management server 20B determines the electronic apparatus30 and the clip terminal 50 that are paired to perform the presencechecking, and enables the electronic apparatus 30 and the clip terminal50 that are paired, to communicate via the clip terminal managementserver 20B (the network 20E). Another method can be used as the methodof connecting the electronic apparatus 30 to the clip terminal 50.

When connection to the clip terminal 50 is made (Act11), the processor30A of the electronic apparatus 30 causes commodity registration to beexecutable. The processor 50A of the clip terminal 50 is connected tothe electronic apparatus (Act40) and starts commodity detection.Commodity detection is a process of detecting objects corresponding tocommodities from an image captured by the camera 50E. The processor 50Adetects the objects corresponding to the commodities from the imagecaptured by the camera 50E through known image processing techniques andstores the position of each object. When the processor 50A detects thatan object has moved from outside the cart to inside the cart or frominside the cart to outside the cart, the processor 50A determines thatthe customer performs an operation of inserting or taking out acommodity. Based on the change in the object location before and afterthis operation, the processor 50A detects that a commodity has beeninserted or taken out. Even when the processor 50A detects an operationby the customer and the number of objects in the cart has not changed,the processor 50A determines that, for example, the customer puts acommodity in the cart again. The processor 50A stores the position ofthe object which has been put in the cart again. The processor 50Astores the object detected from the image captured by the camera 50E ina commodity checklist.

FIG. 8 is a diagram illustrating an example of a commodity checklistaccording to the embodiment. As illustrated in FIG. 8, in the commoditychecklist, data of objects detected from an image captured by the camera50E, data of information regarding commodities that have been registeredin the electronic apparatus 30, and data (for example, D1 to D4)associated with data (check result “OK” or “NG”) indicating a resultobtained by checking whether the registered commodities are identical tothe commodities corresponding to the detected objects, are stored.

The customer registers a commodity by using the camera 30E of theelectronic apparatus 30 to scan a barcode attached to the commoditywhich the customer purchases while shopping in the store (Act12). Theprocessor 30A detects the barcode from the image captured by the camera30E and extracts a commodity code from the barcode. The processor 30Aacquires information regarding the commodity corresponding to thecommodity code from the store server 20A through, for example, the clipterminal management server 20B (the network 20E) and displays theinformation on the touch panel 30D. That is, a commodity name, a price,a commodity image, and the like of the commodity read through thecommodity scanning are displayed so that the customer can confirm thecommodity name, the price, the commodity image, and the like of thecommodity. Here, when the customer performs an operation of instructingcommodity registration (for example, touches a button of the commodityregistration displayed on the touch panel 30D) (Yes in Act13), theprocessor 30A stores the commodity read through the commodity scanningas a registered commodity planned to be purchased (Act14). The processor30A transmits the information regarding the registered commodity to theclip terminal 50 (Act15).

When the clip terminal 50 receives the information regarding theregistered commodity from the electronic apparatus 30 (Yes in Act42),the clip terminal 50 stores the information regarding the registeredcommodity in the commodity checklist (Act43). When an oversizedcommodity is detected with reference to the information regarding theregistered commodity, that is, it is determined that it is not necessaryto insert the commodity in the cart 60 (e.g., the commodity is too largeto be insertable), the processor 50A sets a skip flag in the commoditychecklist (Act45). In the example illustrated in FIG. 8, the skip flagis set as the data D2 in association with a registered commodity“commodity C”. The processor 50A excludes the registered commodity“commodity C” for which the skip flag is set as a checking target. Theprocessor 50A transmits notification of non-necessity of a commodityinsertion into the cart, to the electronic apparatus 30 (Act57).

When the notification of the non-necessity of the commodity insertion isreceived from the clip terminal 50 (Yes in Act18), the processor 30A ofthe electronic apparatus 30 displays a message for notifying thecustomer that it is not necessary to insert the commodity in the cart 60on the touch panel 30D together with the information (the commodityname, the commodity image, the price, and the like) regarding theregistered commodity. The message of the non-necessity of the commodityinsertion may not only be displayed on the electronic apparatus 30 butmay also be displayed on the touch panel 50D of the clip terminal 50.

Conversely, when the registered commodity is not an oversized commodity,the processor 50A acquires, for example, commodity dictionary data foridentifying the registered commodity through image processing, from theclip terminal management server 20B through the network 20E. Thecommodity dictionary data correlates objects to its features so that thecommodity can be recognized from its features such as a barcode attachedto the commodity, a design of a package, or letters, that are detectedfrom the image of the commodity. For example, when the commoditydictionary data corresponding to the registered commodity is not storedin the clip terminal 50, the processor 50A outputs a request fortransmitting the commodity dictionary data corresponding to theregistered commodity to the clip terminal management server 20B. Theclip terminal management server 20B transmits the commodity dictionarydata of the corresponding commodity to the clip terminal 50 in responseto the transmission request from the clip terminal 50. When theelectronic apparatus 30 and the clip terminal 50 communicate through theclip terminal management server 20B and the clip terminal managementserver 20B receives the information regarding the registered commodityfrom the electronic apparatus 30, the commodity dictionary data of thecorresponding commodity may be transmitted to the clip terminal 50 alongwith the information regarding the registered commodity. Alternatively,the commodity dictionary data may be stored in advance in the clipterminal 50.

When the registered commodity is stored in the commodity checklist, theprocessor 50A starts measuring an elapsed time from the storing of theregistered commodity in the commodity checklist. The processor 50Adetermines whether the elapsed time exceeds a preset time before it canbe confirmed that the commodity corresponding to the registeredcommodity has been inserted in the cart 60 (Act46).

The commodity planned to be purchased by the customer is handled in aseries of operations including selecting a commodity from a commodityshelf or the like, scanning the commodity using the electronic apparatus30, and subsequently loading the commodity in the cart 60. Accordingly,when a preset time elapses from the scanning of the commodity, there isa possibility that the customer has forgotten to insert the registeredcommodity in the cart 60. Therefore, the customer is notified. When itis determined that the elapsed time exceeds the preset time for theregistered commodity (Yes in Act46), the processor 50A transmits arequest for inserting the commodity into the cart 60, to the electronicapparatus 30 (Act47).

When the request for inserting the commodity is received from the clipterminal 50 (Yes in Act18), the processor 30A of the electronicapparatus 30 displays a commodity insertion request message promptingthe customer to insert the commodity in the cart 60 along with theinformation (the commodity name, the commodity image, the price, and thelike) regarding the registered commodity on the touch panel 30D (Act19).For example, a message “PLEASE INSERT REGISTERED COMMODITY IN CART.” isdisplayed, as illustrated in FIG. 9A. The commodity insertion requestmessage may be displayed on the electronic apparatus 30 and may also bedisplayed on the touch panel 50D of the clip terminal 50. When there area plurality of commodities which are targets of the commodity insertionrequest message, for example, information regarding the plurality oftarget commodities may be listed to be displayed so that the customercan reliably recognize the target commodities.

When it is detected that the customer performs an operation of insertingor taking out the commodities through image processing performed on animage captured by the camera 50E, the processor 50A of the clip terminal50 detects that the commodities are inserted or taken out based on achange of the objects in the cart before and after the operation isdetected. The processor 50A can detect the operation of inserting ortaking out the commodities not only based on the image but can alsodetermine whether the commodities are inserted or taken out by makingcomparison with a change in a weight detected by the gravimetric sensor601 at a timing at which the operation is detected.

FIG. 10 illustrates an example of an image captured by the camera 50Eaccording to the embodiment. In the example illustrated in FIG. 10, forexample, a state in which three commodities A, B, and E are inserted inthe cart 60 is illustrated.

The processor 50A detects the position of the upper frame 60B of thecart 60 and the region of the basket bottom 60C from an image capturedby the camera 50E. Since the clip terminal 50 is fixed to the cart 60,the regions of the upper frame 60B and the basket bottom 60C in theimage are basically fixed during the shopping. The processor 50Adetermines that the customer performs the operation of inserting ortaking out the commodities, for example, when an object moved to theinside of the cart is detected from the outside the cart using theposition of the upper frame 60B as a reference.

The processor 50A detects objects corresponding to the commodities froman image captured by the camera 50E through known image processing andstores the positions of the objects as object management data.

FIG. 11 is a diagram illustrating examples of objects OJ-A, OJ-B, andOJ-E corresponding to commodities detected from the image illustrated inFIG. 10. The processor 50A detects that the commodities are inserted ortaken out based on a change of the objects in the cart before and afterthe customer inserts or takes out the commodities.

When an object corresponding to a commodity is detected from the image(Yes in Act48), the processor 50A refers to the commodity checklist todetermine whether a registered commodity of the check result “NG” isstored. That is, the processor 50A determines whether there is acommodity for which it is not confirmed that the corresponding commodityis inserted in the cart 60 although the commodity has been registered bythe electronic apparatus 30. Here, when the registered commodity of thecheck result “NG” is stored (Yes in Act49), the processor 50A performsrecognizing an object detected from the image using the commoditydictionary data corresponding to the registered commodity of the checkresult “NG” (Act52).

Here, based on a processing result of the object recognition performedusing the commodity dictionary data, it is determined whether the objectdetected from the image is the registered commodity corresponding to thecommodity dictionary data. For example, when a feature of the image inwhich the object is shown and a feature indicated by the commoditydictionary data indicate similarity exceeding a preset reference value,it is determined that the object is the registered commoditycorresponding to the commodity dictionary data.

Here, when it is determined that the object is the registered commodity(Yes in Act53), the processor 50A stores the object detected from theimage as the detected object in association with the registeredcommodity which corresponds to the commodity dictionary data used in theobject recognition and is previously stored in the commodity checklist(Act54). Data or the like indicating similarity (the degree ofsimilarity) in the object recognition may be stored as data related tothe detected object. Thus, when it is determined that the object is theregistered commodity corresponding to the commodity dictionary data, butthe degree of similarity is low, an attendant can prompt the customer toconfirm the commodity before payment.

The processor 50A sets the check result “OK” corresponding to a pair ofthe registered commodity and the detected object (Act55) and stores astate in which it is confirmed that the registered commodity is insertedin the cart 60. In this case, the processor 50A transmits commodityinsertion confirmation to the electronic apparatus 30 (Act56).

When the commodity insertion confirmation is received from the clipterminal 50 (Yes in Act20) and a commodity insertion request message isdisplayed, the electronic apparatus 30 erases the message from the touchpanel 30D (Act28). Thus, the customer can be notified that the clipterminal 50 confirms that the registered commodity is inserted in thecart 60.

When there are a plurality of registration commodities of the checkresult “NG” in the commodity checklist, the processor 50A performsobject recognition using the commodity dictionary data of the registeredcommodities. For an object which is not determined to be the registeredcommodity through the object recognition (No in Act53), the processor50A stores the object of the check result “NG” as a detected objectunregistered as the commodity in the commodity checklist (Act50). Whenthe object is detected from the image (Yes in Act48) and the registeredcommodity of the check result “NG” has not been set (No in Act49), theprocessor 50A similarly stores the object of the check result “NG” as anunregistered commodity in the commodity checklist (Act50).

The data D3 illustrated in FIG. 8 indicates that the commodity C isregistered in the electronic apparatus 30, but the check result “NG” isset and it is not confirmed that the commodity can be inserted in thecart 60. The data D4 indicates that non-registration commodities E and Fare previously inserted in the cart 60 and are thus detected as objectsE and F (the check result “NG”).

When the detected object (the check result “NG”) as an unregisteredcommodity is stored in the commodity checklist (Act50), the processor50A transmits a commodity registration request to the electronicapparatus 30 (Act51). The processor 30A of the electronic apparatus 30receives the commodity registration request from the clip terminal 50(Yes in Act21) and displays a commodity registration request messageprompting the customer to register the commodity on the touch panel 30D.For example, the commodity registration request message “UNREGISTEREDCOMMODITY IS IN CART. PLEASE REGISTER COMMODITY.” is displayed, asillustrated in FIG. 9B. The commodity registration request message maybe displayed on the electronic apparatus 30 and may also be displayed onthe touch panel 50D of the clip terminal 50 (Act22).

The commodity registration request message displayed on the touch panel50D is erased (Acts 20 and 28) when the commodity is registered, asdescribed above, and it is subsequently confirmed that the commodity isinserted in the cart 60 through the object recognition performed usingthe commodity dictionary data of the registered commodity (Acts 49 to51).

Conversely, when the commodity registered as the commodity planned to bepurchased is returned, the customer scans the barcode attached to thecommodity which is a return target using the camera 30E of theelectronic apparatus 30 (Act12). The processor 30A detects the barcodefrom the image captured by the camera 30E and extracts the commoditycode from the barcode. The processor 30A displays the information (thecommodity name, the price, the commodity image, and the like) regardingthe commodity corresponding to the commodity code on the touch panel 30Dand allows the customer to confirm the commodity which is the returntarget. Here, when the customer performs an operation of instructingcommodity cancellation (for example, touches a button of the commoditycancellation displayed on the touch panel 30D) (No in Act13), theprocessor 30A deletes the commodity read through the commodity scanningfrom the commodities planned to be purchased (Act16). The processor 30Atransmits notification of the registration cancellation to the clipterminal 50 along with information regarding the commodity which is acancellation (return) target (Act17).

When the notification of the registration cancellation is received fromthe electronic apparatus 30 (Yes in Act60), the clip terminal 50determines the corresponding registered commodity from the commoditychecklist based on the information regarding the cancellation targetcommodity (Act61) and deletes the information regarding the registeredcommodity (Act62). The processor 50A transmits notification ofcancellation completion to the electronic apparatus 30 (Act63). When thenotification of the cancellation completion is received (Yes in Act25)and the commodity insertion request message is displayed, the electronicapparatus 30 erases the commodity insertion request message (Act28). Asalready set forth above, when the commodity is registered through thecommodity registration and a setting time subsequently elapses withoutinserting a commodity in the cart 60, the electronic apparatus 30displays the commodity insertion request message. When the customerperforms a commodity cancellation operation on the commodity registeredthrough the commodity registration, the electronic apparatus 30 erasesthe commodity insertion request message for the corresponding commodity.

When the customer takes out the commodity inserted in the cart 60, theprocessor 50A of the clip terminal 50 determines that the customerdetermines an operation of inserting or taking out the commodity basedon an image captured by the camera 50E and detects the objectcorresponding to the commodity taken out through known image processing.As in the case in which the object is detected when the commodity isinserted, a change in a weight is detected by the gravimetric sensor 601to determine that the commodity is taken out.

When it is detected that the object (the commodity) is taken out fromthe cart 60 (Yes in Act64), the processor 50A determines whether theregistered commodity corresponding the detected object is stored in thecommodity checklist (the check result “OK”). Here, when thecorresponding registered commodity is not stored (No in Act65), theprocessor 50A deletes the detection object corresponding to thecommodity which is taken out from the commodity checklist (Act68).

Conversely, when the corresponding registered commodity is stored (Yesin Act65), the processor 50A transmits a commodity cancellation requestto the electronic apparatus 30 (Act66). The processor 50A sets the checkresult “NG” in the detection object corresponding to the commodity whichis taken out (Act67).

When the commodity cancellation request is received from the clipterminal 50 (Yes in Act23), the processor 30A of the electronicapparatus 30 displays a commodity cancellation request message promptingthe customer to perform an operation of cancelling the commodity on thetouch panel 30D (Act24). For example, as illustrated in FIG. 9C, thecommodity cancellation request message “COMMODITY HAS BEEN TAKEN OUTFROM CART. PLEASE PERFORM RETURN OPERATION WHEN COMMODITY IS RETURNED.”is displayed, as illustrated in FIG. 9C. The commodity cancellationrequest message may be displayed on the electronic apparatus 30 and mayalso be displayed on the touch panel 50D of the clip terminal 50.

When the commodity is taken out from the cart 60, the data stored in thecommodity checklist is immediately updated. Alternatively, after apreset time (for example, 10 seconds) has elapsed since the commodity istaken out from the cart 60, the data stored in the commodity checklistmay be updated as data for which there is a high possibility of thecommodity being cancelled (returned). In other words, the commoditycancellation request message may be immediately displayed on theelectronic apparatus 30 in response to extraction of the commodity ormay be displayed after the set time is elapsed.

In this way, when the customer mounts the clip terminal 50 on the cart60 and performs shopping while performing the commodity registrationusing the electronic apparatus 30, the customer can easily confirm themessage displayed on the electronic apparatus 30 although the customerforgets to register the commodity planned to be purchased and forgets toinsert the commodity in the cart 60.

When the registration of the commodity planned to be purchased iscompleted, for example, the customer operates a subtotal button to givenan instruction to end the commodity registration. When the instructionto end the commodity registration is given by the customer (Yes inAct26), the processor 30A of the electronic apparatus 30 displays aconfirmation screen of the registered commodities and allows thecustomer to designate a payment method (e.g., credit, cash, directdebit, electronic money, or the like). The processor 30A generatespayment information based on information relating to the commoditiesregistered through the commodity registration.

The processor 30A of the electronic apparatus 30 transmits thenotification of end of the registration to the clip terminal 50 (Act27).When the notification of the end of the registration is received fromthe electronic apparatus 30 (Yes in Act69), the processor 50A of theclip terminal 50 transmits data of the commodity checklist to theelectronic apparatus 30 (Act70). The data also includes, for example,data indicating the degree of similarity at the time of recognizing thedetected objects. The processor 5 OA transmits the data of the commoditychecklist, and subsequently clears the data of the commodity checklistto end the process (Act71).

When an instruction to end the registration is given and the data of thecheck result “NG” remains in the data stored in the commodity checklist,a message prompting the customer to confirm the data, is displayed againon the touch panel 30D of the electronic apparatus 30 (and optionally,also on the touch panel 50D of the clip terminal 50).

When a final confirmation is given from the customer, the processor 30Aof the electronic apparatus 30 displays a code (a barcode or a2-dimensional code) that includes payment information and data promptingthe attendant to confirm the final confirmation.

The attendant scans the code displayed on the touch panel 30D of theelectronic apparatus 30 using the camera of the electronic apparatus 40.The electronic apparatus 40 displays the payment information included inthe code and a confirmation screen including the data which is necessaryto be confirmed. For example, a list (including commodity names, prices,subtotal amount, and the like) of the commodities planned to bepurchased and registered through the registration by the customer andreward points are displayed on the electronic apparatus 40. Thus, theattendant can confirm whether the commodities inserted in the cart 60are identical to the registered commodities. Further, the electronicapparatus 40 displays, for example, the commodities corresponding to thedetected objects of which the degree of similarity is low in the objectrecognition carried out by the clip terminal 50, based on the data whichis necessary to be confirmed. Thus, the attendant can easily recognizethe commodities which is particularly necessary to be confirmed, andthus it is possible to reduce a burden on the confirmation work.

As described above, for example, the image processing in the clipterminal 50 is used to check whether the registration commodities areidentical to the commodities inserted in the cart 60. However, theweights of the commodities detected by the gravimetric sensor 601 may beused to check the commodities. For example, since data of the weights ofthe commodities is included in data regarding the commodities providedfrom the store server 20A, the checking can be performed based onwhether the weight increase at the time of inserting a commodity in thecart 60 is identical to the weight of the registered commodity within anacceptable error range. In particular, using data of weights detected bythe gravimetric sensor 601 is effective in checking sell-by-weight typecommodities. For example, it is possible to confirm a dishonest actionof inserting a commodity heavier than the same kind of registeredcommodity in the cart 60 after a commodity having a less weight isregistered through the registration. When there is a sell-by-weight typecommodity in the commodity checklist and a difference between the weightof a registered commodity and a weight at the time of insertion in thecart 60 is greater than a set value, data indicating that confirmationis necessary is stored. Thus, as described above, the attendant caneasily recognize that the confirmation is particularly necessary.

Further, as described above, it is checked whether the commoditiesregistered in the registration are identical to the commodities insertedin the cart 60. However, the clip terminal 50 can also be used to checkthe registered location of commodities. For example, when a differencebetween a location of a target commodity in a store at a timing ofcommodity registration in the electronic apparatus 30 (or a timing ofdetecting a commodity by the clip terminal 50) and a position of acommodity shelf on which the target commodity is located in the store isgreater than a predetermined distance, it can also be determined thatconfirmation is necessary.

FIG. 12 is a flowchart illustrating registration location determinationperformed to determine whether confirmation is necessary, according tothe embodiment. The processor 50A of the clip terminal 50 performs theregistration location determination based on the check program. Theregistration location determination is performed in parallel with theabove-described presence checking.

When commodity information regarding a commodity that has beenregistered, is received from the electronic apparatus 30 (Yes in Act80)or when it is confirmed that the registered commodity corresponds to thedetected object (the check result “OK”) (Yes in Act81), the processor50A detects a current position of the commodity (Act82). The currentposition can be detected, for example, by communicating with an RF tagattached to each of commodity shelves in the store by the reader-writer50J.

FIG. 13 illustrates an example of layout of commodity shelves 70-1 to70-5 and a position of the cart 60 in the store. The clip terminal 50mounted on the cart 60 can determine that the cart 60 is near thecommodity shelf 70-5 when it is possible to communicate with an RF tag72-5 mounted on the commodity shelf 70-5 through communication within arestricted range by the reader-writer 50J. The current position of theclip terminal 50 (and thus indirectly the position of the cart 60) canbe detected using not only the method of using the reader-writer 50J(the RF tag) but also a method such as a difference in a radio waveintensity between the communication module 50F and the plurality ofaccess points 20F or image processing performed on an image obtained byimaging the inside of the store to detect the position of the cart 60within the store.

When the current position is detected, the processor 50A acquires dataindicating the commodity position of a registered commodity that islater sold through, for example, the store server 20A or the electronicapparatus 30 (Act83). The processor 50A determines whether a differencebetween the current position and the commodity position (the position ofthe commodity shelf) is greater than a predetermined distance (Act84).When a commodity 74 disposed on the commodity shelf 70-1 illustrated inFIG. 13 is registered or is inserted in the cart 60 at a position closeto the commodity shelf 70-5, it is determined that confirmation isnecessary since the customer performs an operation different from anormal operation (Yes in Act85). The clip terminal 50 storesconfirmation data in association with data corresponding to the detectedobject and the corresponding registered commodity to the commoditychecklist. Thus, as described above, the attendant can easily recognizethat the confirmation is particularly necessary.

According to at least one of the above-described embodiment, the clipterminal 50 is used to check whether the commodity informationregistered through the registration by the electronic apparatus 30 isidentical to the commodities corresponding to the detected objectscorresponding to the commodities inserted in or taken out from the cart60 and detected by the clip terminal 50. According to the check result,a message prompting the customer to confirm the data can be displayed.Therefore, it is possible to reliably register the commodities plannedto be purchased through a customer operation.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

The processes described in the above-described embodiments can beimplemented as a program executed by a computer on a recording mediumsuch as a magnetic disk (a flexible disc, a hard disk, or the like), anoptical disc (a CD-ROM, a DVD, or the like), or a semiconductor memoryto be provided to various apparatuses. The program may be transmittedthrough a communication medium to be provided to various apparatuses. Acomputer reads the program recorded on a recording medium or receivesthe program via a communication medium and an operation is controlled bythe program to perform the above-described processes.

What is claimed is:
 1. A computing device installed in or mountable to ashopping cart, comprising: a camera configured to capture an image; anda processor configured to: acquire commodity data of a registeredcommodity, when an object is placed in the shopping cart, perform objectrecognition on an image of the object captured by the camera, anddetermine whether the recognized object is a commodity corresponding tothe acquired commodity data, and upon determining that the recognizedobject is not the commodity corresponding to the acquired commoditydata, issue a request for displaying a message instructing registrationof the recognized object as a registered commodity.
 2. The computingdevice according to claim 1, further comprising: a display, wherein themessage is displayed on the display.
 3. The computing device accordingto claim 1, further comprising: a network interface, wherein the requestis transmitted to another device via the network interface.
 4. Thecomputing device according to claim 1, further comprising: a networkinterface, wherein the commodity data is received from another devicevia the network interface.
 5. The computing device according to claim 4,wherein said another device is a wireless terminal configured to displayinformation about registered commodities.
 6. The computing deviceaccording to claim 1, further comprising: an interface connectable to asensor configured to weigh objects in the shopping cart, wherein theprocessor is configured to perform the object recognition upon detectionof changes in weight of the objects in the shopping cart by the sensor.7. The computing device according to claim 1, wherein an imaging area ofthe camera includes an entire perimeter of the shopping cart.
 8. Thecomputing device according to claim 1, further comprising: a wirelesstag reader having a communication range to detect a wireless tagattached to an object placed in the shopping cart.
 9. The computingdevice according to claim 1, further comprising: a wireless tag readerhaving a communication range to detect a wireless tag attached to anobject displayed on a commodity shelf.
 10. The computing deviceaccording to claim 1, wherein the computing device is mounted on ahandle of the shopping cart.
 11. A method performed by a computingdevice installed in or mountable to a shopping cart, the methodcomprising: acquiring commodity data of a registered commodity; when anobject is placed in the shopping cart, capturing an image of the object,performing object recognition on the image, and determining whether therecognized object is a commodity corresponding to the acquired commoditydata; and upon determining that the recognized object is not thecommodity corresponding to the acquired commodity data, issuing arequest for displaying a message instructing registration of therecognized object as a registered commodity.
 12. The method according toclaim 11, further comprising: displaying the message on a display. 13.The method according to claim 11, further comprising: transmitting therequest to another device via a network interface.
 14. The methodaccording to claim 11, further comprising: receiving the commodity datafrom another device via a network interface.
 15. The method according toclaim 14, wherein said another device is a wireless terminal configuredto display information about registered commodities.
 16. The methodaccording to claim 11, wherein the object recognition is performed upondetection of changes in weight of objects in the shopping cart by asensor.
 17. The method according to claim 11, wherein an imaging area ofthe camera includes an entire perimeter of the shopping cart.
 18. Themethod according to claim 11, further comprising: detecting a wirelesstag attached to the object placed in the shopping cart by a wireless tagreader.
 19. The method according to claim 11, further comprising:detecting a wireless tag attached to an object displayed on a commodityshelf by a wireless tag reader.
 20. The method according to claim 19,wherein a location of the shopping cart can be determined based oninformation acquired from the detected wireless tag.